Skip to content

[branch-54] Fix: compact view buffers in ScalarValue::compact for all container types (#21934)#22446

Open
alamb wants to merge 1 commit into
apache:branch-54from
alamb:alamb/backport_21934
Open

[branch-54] Fix: compact view buffers in ScalarValue::compact for all container types (#21934)#22446
alamb wants to merge 1 commit into
apache:branch-54from
alamb:alamb/backport_21934

Conversation

@alamb
Copy link
Copy Markdown
Contributor

@alamb alamb commented May 22, 2026

This PR cherry-picks the fix for ScalarValue::compact to compact view buffers for all container types onto branch-54.

apache#21934)

## Which issue does this PR close?

- Closes apache#21928.

## Rationale for this change

ScalarValue::compact copies array data for container types to release
sliced-buffer overhead, but never called .gc() on nested StringViewArray
/ BinaryViewArray. A scalar extracted from a large batch would therefore
still hold a reference to the entire original view backing buffer,
negating the benefit of compaction for any list, struct, or map whose
values are view-typed.
## What changes are included in this PR?

- ScalarValue::compact now compacts nested view buffers for all
container types (List, LargeList, FixedSizeList, ListView,
LargeListView, Struct, Map), trimming StringViewArray / BinaryViewArray
backing buffers to only the
  referenced bytes.
- The internal compact_view_buffers helper recursively handles
FixedSizeList, ListView, LargeListView, and Map.

## Are these changes tested?

new unit tests are added in scalar::tests, one per container type. Each
test verifies that after compact() the backing buffer is reduced to
exactly the bytes of the referenced string, and that the scalar value is
preserved.

## Are there any user-facing changes?

No. The public compact / compacted API is unchanged; this PR only fixes
the behaviour for view-typed nested arrays.

Co-authored-by: Bert Vermeiren <bert.vermeiren@datadobi.com>
Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants